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Abstract. Robots assisting humans in complex domains have to represent knowl¬ 
edge and reason at both the sensorimotor level and the social level. The architec¬ 
ture described in this paper couples the non-monotonic logical reasoning capabil¬ 
ities of a declarative language with probabilistic belief revision, enabling robots 
to represent and reason with qualitative and quantitative descriptions of knowl¬ 
edge and degrees of belief. Specifically, incomplete domain knowledge, including 
information that holds in all but a few exceptional situations, is represented as a 
Answer Set Prolog (ASP) program. The answer set obtained by solving this pro¬ 
gram is used for inference, planning, and for jointly explaining (a) unexpected 
action outcomes due to exogenous actions and (b) partial scene descriptions ex¬ 
tracted from sensor input. For any given task, each action in the plan contained 
in the answer set is executed probabilistically. The subset of the domain relevant 
to the action is identified automatically, and observations extracted from sensor 
inputs perform incremental Bayesian updates to a belief distribution defined over 
this domain subset, with highly probable beliefs being committed to the ASP pro¬ 
gram. The architecture’s capabilities are illustrated in simulation and on a mobile 
robot in the context of a robot waiter operating in the dining room of a restaurant. 


1 Introduction 

Robots collaborating with humans in complex domains receive far more raw sensor 
data than can be processed in real-time. The information extracted from the sensor 
inputs can be represented probabilistically to quantitatively model the associated un¬ 
certainty (“90% certain I saw the book on the shelf”). Robots also receive useful com- 
monsense knowledge that is difficult to represent quantitatively (“books are usually in 
the library”), and human participants may not have the time and expertise to provide 
elaborate and accurate feedback. To collaborate with humans, these robots thus need 
to represent knowledge and reason at both the cognitive level and the sensorimotor 
level. This objective maps to fundamental research challenges in knowledge represen¬ 
tation and reasoning. The architecture described in this paper exploits the complemen¬ 
tary strengths of non-monotonic logical reasoning and probabilistic belief revision as 
a significant step towards addressing these challenges. Specifically, the commonsense 
logical reasoning capabilities of Answer Set Prolog (ASP), a declarative language, is 
coupled with probabilistic belief updates, to support the following key features: 

• An ASP program represents incomplete domain knowledge, including information 
that holds in all but a few exceptional situations. The answer set obtained by solv¬ 
ing the ASP program is used for planning and jointly (a) explaining unexpected 



action outcomes by reasoning about exogenous actions; and (b) identifying object 
occurrences that best explain partial scene descriptions obtained from sensor inputs. 

• For any given task, each action in the plan created by inference in the ASP program 
is executed probabilistically. The relevant subset of the domain (for this action) is 
identified automatically, and the sensor observations perform incremental Bayesian 
updates to a belief distribution defined over this subset of the domain, committing 
highly probability beliefs as statements to the ASP program. 

The architecture thus enables robots to represent and reason with qualitative and quan¬ 
titative descriptions of knowledge and degrees of belief. In this paper, the architecture’s 
capabilities are demonstrated in simulation and on a mobile robot, in the context of a 
robot waiter operating in the dining room of a restaurant. 

2 Related Work 

Knowledge representation, planning and explanation generation are well-researched ar¬ 
eas in robotics and artificial intelligence. Logic-based representations and probabilistic 
graphical models have been used to plan sensing, navigation and interaction for robots 
and agents. Formulations based on probabilistic representations (by themselves) make 
it difficult to perform commonsense reasoning, while classical planning algorithms and 
logic programming tend to require considerable prior knowledge of the domain and the 
agent’s capabilities, and make it difficult to merge new, unreliable information with an 
existing knowledge base. For instance, the non-monotonic logical reasoning capabilities 
of ASP El have been used for tasks such as reasoning by simulated robot housekeep¬ 
ers El and coordination of robot teams El. However, ASP does not support proba¬ 
bilistic analysis of uncertainty, whereas a lot of information extracted from sensors and 
actuators on robots is represented probabilistically. 

Approaches for generating explanations (e.g., through abductive inference or plan 
diagnosis) use the systems description and observations of system behavior to explain 
unexpected symptoms ED, or use weaker system descriptions and depend on heuris¬ 
tic representation of intuition and past experience II6I8L Probabilistic and first-order 
logic-based representations have been combined for better abductive inference flU. 
Researchers have also designed architectures for robots that combine deterministic and 
probabilistic algorithms for task and motion planning Q, combine declarative program¬ 
ming and continuous-time planners for path planning in robot teams im, or combine 
a probabilistic extension of ASP with partially observable Markov decision processes 
(POMDPs) for human-robot dialog CSl. Some principled algorithms that combine log¬ 
ical and probabilistic reasoning include Markov logic network ifTOl . Bayesian logic Qj 
and probabilistic extensions to ASP lfT]5l . However, algorithms based on first-order 
logic do not provide the desired expressiveness for modeling uncertainty, e.g., it is not 
always possible to express degrees of belief quantitatively. Algorithms based on logic 
programming do not support one or more of the desired capabilities such as incremental 
revision of (probabilistic) information; reasoning as in causal Bayesian networks; and 
reasoning with large probabilistic components. Towards addressing these challenges, 
our prior work developed architectures that couple declarative programming and proba¬ 
bilistic graphical models for logical inference, deterministic and probabilistic planning 
on robots mM- This paper retains the coupling between logical and probabilistic 
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Fig. 1. An overview of the architecture that combines the complementary strengths of declarative 
programming and probabilistic graphical models for inference, planning, and diagnosis. 


reasoning but significantly expands the capabilities of these architectures to support; 
(1) explanation of unexpected action outcomes and the partial descriptions extracted 
from sensor inputs; and (2) representation and reasoning at a higher resolution using 
ASP, making the probabilistic reasoning more computationally efficient. 


3 Proposed Architecture 

Figure[T]is an overview of the mixed architecture. The symbolic representation is trans¬ 
lated to an Answer Set Prolog (ASP) program used for non-monotonic logical inference 
and planning a sequence of actions for any given task. For each action, the relevant sub¬ 
set of the domain is defined automatically. Sensor observations perform incremental 
Bayesian updates to a probability distribution over this domain subset, committing high 
probability beliefs (action outcomes, observation of object attributes) as statements to 
the ASP program. Observed unexpected action outcomes are explained by reasoning 
about exogenous actions, and objects are identified to best explain the partial descrip¬ 
tions extracted from visual cues. ASP-based representation and reasoning is performed 
at a resolution that provides high reliability while also simplifying the (coupled) prob¬ 
abilistic reasoning and tailoring it to specific actions. 

The syntax, semantics and representation of the transition diagrams of the archi¬ 
tecture’s domain representation are described in an action language AL IJI. AL has 
a sorted signature containing three sorts: statics, fluents and actions. Statics are do¬ 
main properties whose truth values cannot be changed by actions, fluents are properties 
whose values are changed by actions, and actions are elementary actions that can be 
executed in parallel. AL allows three types of statements: 

a causes 4 (Causallaw) 

/ if po,... , Pm (State constraint) 

impossible ao,...,ak if po,...,pm (Executability condition) 

where a is an action, I is a literal, 4 is a basic fluent (also called inertial fluent) literal, 
and pQ,...,pm are domain literals (any domain property or its negation). A collection 
of statements of AL forms a system description. 













As an illustrative example used in this paper, consider a robot waiter that greets 
and seats people at tables in a restaurant, and delivers orders. The sorts of the domain 
are arranged hierarchically, e.g., location and thing are subsorts of entity, animate and 
inanimate are subsorts of thing', person and robot are subsorts of animate; object is 
a subsort of inanimate; and room, area, door, and floor are subsorts of location. We 
include specific rooms, e.g., kitchen and dining, and consider objects of sorts such as 
table, chair and plate, to be characterized by attributes size, color, shape, and location. 
The sort step is included for temporal reasoning. 

ASP Domain Representation: The ASP program is based on a domain representa¬ 
tion that includes a system description and a history with defaults has 

a sorted signature that defines the names of objects, functions, and 

predicates available for use, and axioms that describe a transition diagram T//. Fluents 
are defined in terms of the sorts of their arguments, e.g., has Jocationfhing, location) 
inJiand{robot,object), isjopen{door), and canjnove{robot,location). The first three 
are basic fluents that obey the laws of inertia and can be changed directly by actions; the 
last one is a defined fluent that is not subject to inertia and cannot be changed directly by 
an action. Statics such as connected{location, location) and belongs{location, location) 
specify connections between locations, relation holds{fluent,step) implies a specific 
fluent holds at a specific timestep, and occurs {act ion, step) (hypothesizes) that a spe¬ 
cific action occurs at a specific timestep. We include actions, e.g., move {robot, location), 
seat_person{robot,person,table), search_person{robot,area), pickup{robot,object), 
putdown{robot,object), and define domain dynamics using causal laws such as: 

move{R,L) causes hasJocation{R,L) (1) 

pickup{R,0) causes inJiand{R,0) 
open{R,D) causes is-open{D) 

seat-person{R,P,T) causes hasJocation{P,L) if has Jocation{T,L) 
state constraints such as: 

hasJocation{0,L) if hasJocation{R,L), inJiand{R,0) (2) 

-^hasJocation{Th,L 2 ) if hasJocation{Th,L\),L\fL 2 
hasJocation{Th,L 2 ) if hasJocation{Th,L\), belongs{Li,L 2 ) 
canjnove{R,L 2 ) if hasJocation{R,L\), connected{Li,L 2 ) 

and executability conditions such as: 

impossible move{R,L) if hasJocation{R,L) (3) 

impossible pickup{R,0) if hasJocation{R,L\),hasJocation{0,L2),L\fL2 
impossible open{R,D) if is_open{D) 

Since robots frequently receive default domain knowledge that is true in all but a few 
exceptional situations, the domain history in addition to hpd{action, step) and 
obs{fluent,boolean,step), the occurrence of specific actions and the observation of 
specific fluents at specific time steps, contains prioritized defaults describing the values 
of fluents in their initial states. For instance, it may be initially believed that dishes to be 
delivered are typically on a table between the kitchen and the dining room—if they are 



not there, they are still in the kitchen. Existing definitions of entailment and consistency 
are used to reason with such histories, and any observed exceptions M- 

The domain representation is translated into a program in CR-Prolog 

that incorporates consistency restoring rules in ASP 0. n includes the causal laws 
of inertia axioms, closed world assumption for actions and defined fluents, reality 
checks, and records of observations, actions and defaults from Every default is 
turned into an ASP rule and a consistency-restoring (CR) rule that allows us to assume 
the default’s conclusion is false to restore H’s consistency. ASP is based on stable model 
semantics, introduces concepts such as default negation and epistemic disjunction, and 
represents recursive definitions, defaults, causal relations, and language constructs that 
are difficult to express in classical logic formalisms. The ground literals in an answer 
set obtained by solving FI represent beliefs of an agent associated with FI —statements 
that hold in all such answer sets are program consequences. Inference and planning 
can be reduced to computing answer sets of program FF by adding a goal, a constraint 
stating that the goal must be achieved, and a rule generating possible future actions. 

Our architecture supports reasoning about exogenous actions to explain the unex¬ 
pected (observed) outcomes of actions. Eor instance, to reason about a door between 
the kitchen and the dining room being locked by a human, and to reason about a person 
moving away from a known location, we introduce exogenous actions locked{door) 
and moved-from{person, location) respectively, and suitably add (or revise) axioms: 

is-open{D) •(— open{R,D), ^abiD) (4) 

ab{D) •<— locked{D) 

^hasJocation{P,L) moved-from(P,L), hasJocation{P,L) 

where a door is considered abnormal if it has been locked, say by a human. We also 
introduce an explanation generation rule and a new relation expl: 

occurs{A,I) I ^ occurs{A,I) ^ exogenous_action{A), I < n (5) 

expl(AJ) ^ action(exogenous,A), occurs(AJ)^ not hpd{A,I) 

where expl holds if an exogenous action is hypothesized but there is no matching record 
in the history. We also include awareness axioms and reality check axioms: 

holds{F,0) or ^ holds{F,0) •<— fluent [basic ^F) % awareness axiom (6) 

occurs [A, I) ^ hpd[A,I) 

•<— obs[fluentfrue,I),^holds[fluent,I) % reality check 

•<— obs[fluent,false,I),holds[fluent,I) % reality check 

The reality check axioms cause a contradiction when observations do not match ex¬ 
pectations, and the explanation for such unexpected symptoms can be extracted from 
the answer set of the corresponding program 0. The new knowledge is included and 
used to generate the subsequent plans. This approach provides all explanations of an 
unexpected symptom—using a CR rule instead of the explanation generation rule (in 
Statement]^ provides the minimal explanation (see below). 

A robot processing sensor inputs (e.g., camera images) is typically able to extract 
partial descriptions of scene objects. The proposed architecture also identifies object 


occurrences that best explain these partial descriptions. In our illustrative example, we 
introduce static relations to establish object class membershifQand introduce relations 
to capture ideal (and default) definitions of object attributes, e.g., for a table; 

has-Color{0,white) •<— member{0,table) (7) 

has-size{0,medium) ^ member{0, table) 

hasjwheels{4) -(r- member{0,table), ^ hasJocation(0,kitchen) 

where tables usually have wheels except in the kitchen. Similarly, for a chair: 

has_color{0, white) member{0, chair) ( 8 ) 

has_size{0,medium) ^r- member{0,chair) 

-^hasjwheels(0) ^ memberiO,chair) 


Other objects and object attributes are encoded similarly. As before, a reality check 
axiom causes an inconsistency when an object does not have a class label due to incom¬ 
plete information, and a CR rule restores consistency by assigning class labels: 

object{0), not class_known{0) % reality check (9) 

is_a{0,C) object{0) % CR rule 


This assignment of a class label to an object is based on the smallest number of rules 
that need to be relaxed to support the assignment. This information is also added to the 
ASP program and be used for subsequent reasoning. However, both planning and object 
recognition are based on processing sensor inputs and moving to specific locations— 
these tasks are accomplished using probabilistic algorithms, as described below. 


Probabilistic Domain Representation: Our previous work created the ASP-based rep¬ 
resentation at a coarser resolution (e.g., rooms and places), and refined it by adding 
suitable actions, fluents and sorts (e.g., cells in rooms) to define a transition diagram 
and create its probabilistic version that was modeled as a POMDP ifTSIl . The proposed 
architecture models the domain at a higher resolution using ASP. For any given task, the 
ASP-based plan consists of primitive actions that can be executed by the robot. Each 
such action is executed probabilistically, with the robot maintaining a probability (be¬ 
lief) distribution over the relevant subset of the domain that is identified automatically 
based on the set of related fluents, e.g., for moving between two tables, the robot only 
needs to reason about its own location in a subset of areas. The belief distribution is 
revised incrementally by sensor observations using Bayesian updates. For instance, to 
update the belief about the location of a specific dish in the dining room; 

p{Oi\Ei)p{Ei) 


p{Ei\Oi) = 


{p{Oi\Ei)p{Ei)+p{Oi\-^Ei)p{-^Ei)) 


( 10 ) 


where (9, is the event the dish was observed in area /, £, is the event the dish exists in 
area i, making p((9,j£’,) and p(£’,j6>,) are the observation likelihood and the posterior 

* Relation member(object,class) is applied recursively in a class hierarchy, is_a{object,class) 
denotes an instance of a specific class, and class_known{object) holds for any object whose 
class label is known. 




probability of existence of the dish in each area in the dining room. The initial knowl¬ 
edge (i.e., prior: p{Ei) is based on domain knowledge or statistics collected in an initial 
training phase (see Section |^. A Bayesian state estimation approach is used by the 
robot to estimate its own position (e.g., particle filters), navigate, and to process sensor 
inputs to extract information about objects being observed. 

In summary, for any given task, ASP planning provides a plan with deterministic ef¬ 
fects. The hrst action in the plan and relevant information (from ASP inference) identify 
the subset of the domain to be represented probabilistically, and set the initial (proba¬ 
bilistic) belief distribution. The action is executed probabilistically, updating the belief 
distribution until a high belief indicates action completion or a time limit is exceeded, 
and adding relevant statements to the ASP program. Any unexpected action outcomes 
and partial scene descriptions are explained by reasoning about exogenous actions and 
possible objects. Once these explanations restore consistency, either the next action (in 
the plan) is selected for execution or a new plan is created. In what follows, we refer 
to the proposed architecture as the “mixed architecture”, and compare it with two algo¬ 
rithms: (1) ASP-based reasoning for completing the assigned tasks; and (2) a (greedy) 
probabilistic approach that maintains a probabilistic belief distribution and heuristically 
selects actions (and makes decisions) based on the most likely state. 

The mixed architecture raises some subtle issues. First, committing probabilistic 
beliefs above a specihc threshold (e.g., 0.85) as fully certain statements to the ASP pro¬ 
gram may introduce errors, but the non-monotonic reasoning capability of ASP helps 
the robot recover. Second, with previous work that reasoned at a coarser resolution with 
ASP and used POMDPs for probabilistic planning (a) computing POMDP policies for 
each ASP action is computationally expensive; and (b) there may be improper reuse 
of information if the probabilistic belief distribution is not reset between trials m. 
Third, the mixed architecture presents an interesting trade-off between the resolution 
of symbolic representation and probabilistic representation. Moving most of the rea¬ 
soning to a symbolic representation can reduce accuracy and also be computationally 
expensive—the mixed architecture is a good trade-off of accuracy and efficiency. 

4 Experimental Setup and Results 

Experiments were conducted in simulation and on a mobile robot in scenarios that 
mimic a robot waiter in a restaurant. The robot’s tasks include finding, greeting and 
seating people at tables, and delivering orders appropriately. 


4.1 Experimental Setup and Hypotheses 

The experimental trials used existing implementations of relevant control and sensor 
input processing algorithms. In an initial training phase, the robot collected statistics of 
executing these algorithms to compute the motion error models and observation likeli¬ 
hood models. These models were also used to make the simulation trials more realistic. 

The experimental trials considered three hypotheses, evaluating that the mixed ar¬ 
chitecture: (HI) generates plans for different tasks, and explains unexpected outcomes 
and partial descriptions extracted from sensor inputs; (H2) significantly improves the 
task completion accuracy and provides similar task completion time in comparison with 


just ASP-based reasoning; and (H3) significantly improves task completion time and 
provides similar accuracy in comparison with the purely probabilistic approach. We 
provide both qualitative and quantitative results in simulation and on a mobile robot. 

4.2 Experimental Results 

The following execution traces demonstrate the planning and diagnosis capabilities. 
Example 1 [Explain unexpected action outcome] 

The task is to return dish ds\ to the kitchen from the dining room, e.g., from tablel to 
area 03 in Figure [2(a)| Unknown to the robot, the door d 2 has been locked. 

• The robot is in the dining room with the dish in hand: 

holds{hasJocation{robot,dining),0), hold{hasJocation{robot,areal),0), 
holds{inJiand {robot,ds\), 0) 

The initial plan obtained by computing the answer set is: 

occurs{move{robot, 02 ), 1 ), occurs{move{robot,d 2 ),2), 
occurs{o pen{robot ,d2),3), occurs {move {robot , 013 ), 4 ), 
occurs{putdown{robot ,dsi),5). 

• Each step in this plan, starting with the first one, is executed probabilistically. 

• Unfortunately, the robot’s attempt to open door d 2 does not produce the expected 
observation—instead, obs{is_open{d 2 ), false,3) is added to the history. 

• Diagnosis provides the explanation expl{locked{d 2 ),3), which invokes Statement]^ 
to restore consistency. 

• The robot seeks human help to unlock the door, before creating a new plan to 
successfully return the dish to the kitchen. 

Example 2 [Explain partial scene description] 

The robot delivering a dish sees a medium-sized white object from a distance, but is 
unable to assign a class label in the absence of any further information. 

• Then initial knowledge consists of: 
hasjize{ob\,medium), hasj:olor{ob\,white) 

• Two possible interpretations are generated by Statements |7|8| as an explanation: 
isua{obi,table) or isM{ob\,chair). 

• As the robot gets closer, it observes: has .wheel {obi, A), has Jocation{obi,dining), 
i.e., the object has wheels and is in the dining room. Statements |7|8 [ provide the in¬ 
terpretation: is M{ob\, table). 

Simulation Experiments: The robot in the simulator had to find and move (seat) dishes 
(people) to specific locations. Table[T]summarizes the results—each entry is the average 
of 500 trials. The task, initial position of the robot, and position of objects, were differ¬ 
ent between the trials, and paired trials were used to establish statistical significance. In 
each paired trial, for each approach being compared, the initial location of the robot and 
the location of domain objects are the same. The statistically significant results show 
that the mixed architecture (a) significantly improves the accuracy and provides similar 
task completion time in comparison with ASP-based reasoning; and (b) significantly 
improves task completion time and provides similar accuracy in comparison with the 




Table 1. Task completion accuracy and time using only ASP, and using a probabilistic approach, 
expressed as a factor of the values provided by the mixed architecture. The proposed mixed 
architecture significantly reduces the task completion time while improving the accuracy. 


Algorithms 

Evaluation metrics 

Accuracy 

Time 

ASP only 

0.82 

1.06 ±0.6 

Probabilistic 

0.99 

3.32±3.0 

Mixed approach 

1 

1 



(a) Example domain. (b) Turtlebot robot. 


Fig. 2. (a) Example map of illustrative domain used for experimental evaluation, with rooms, 
doors, and tables (people and robot not shown); and (b) the Turtlebot mobile robot platform. 


probabilistic approach. Furthermore, the planning and execution time are significantly 
reduced in comparison with approaches that combined ASP with probabilistic graphi¬ 
cal models |IT3]| . while providing comparable accuracy—analysis in other domains may 
help automate the choice of resolution for symbolic and probabilistic representations 
for a given task and domain. 


Robot Experiments: Trials were conducted on a Turtlebot (Figure [2(b^ equipped with 
a Kinect (RGB-D) sensor, range sensors, and an on-board processor running Ubuntu 
Linux. Our architecture and algorithms were implemented using the Robot Operating 
System (ROS). Trials included instances of the domain introduced in Section]^ each 
with one or more tables, people and other objects, e.g., Figure 2(a) The robot was 
equipped with probabilistic algorithms to determine the attribute values of objects (e.g., 
color and shape) from camera images, revise the map of the domain, and determine its 
location in the map. The robot was able to use the mixed architecture to successfully 
complete the assigned tasks in all such scenarios, with results of paired trials being sim¬ 
ilar to those obtained in simulation. A video of an experimental trial showing planning 
and diagnosis can be viewed online: https: //vimeo. com/130279856 


5 Conclusions 

This paper described an architecture that mixes the complementary strengths of declara¬ 
tive programming and probabilistic belief updates. Plans created using ASP-based non¬ 
monotonic logical reasoning are implemented probabilistically, with high probability 
observations and action outcomes adding statements to the ASP program. The architec¬ 
ture enables a robot to explain unexpected action outcomes by reasoning about exoge¬ 
nous actions, and to identify objects that best explain partial scene descriptions. These 
capabilities have been demonstrated through experimental trials in simulation and on 
















































a mobile robot in scenarios that mimic a robot waiter in a restaurant’s dining room. 
Future work will further investigate the tight coupling and transfer of control between 
the logical and probabilistic representations, with the long-term objective of enabling 
robots to collaborate with humans in complex application domains. 
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